Aiming at the problems of low utilization rate, high error rate, no recognition ability and manual participation in video surveillance processing in coastal and inland waters of China, a new ship tracking and recognition method based on Darknet network model and YOLOv3 algorithm was proposed to realize ship tracking and real-time detection and recognition of ship types, solving the problem of ship tracking and recognition in important monitored waters. In the Darknet network of the proposed method, the idea of residual network was introduced, the cross-layer jump connection was used to increase the depth of the network, and the ship depth feature matrix was constructed to extract advanced ship features for combination learning and obtaining the ship feature map. On the above basis, YOLOv3 algorithm was introduced to realize target prediction based on image global information, and target region prediction and target class prediction were integrated into a single neural network model. Punishment mechanism was added to improve the ship feature difference between frames. By using logistic regression layer for binary classification prediction, target tracking and recognition was able to be realized quickly with high accuracy. The experimental results show that, the proposed algorithm achieves an average recognition accuracy of 89.5% with the speed of 30 frame/s; compared with traditional and deep learning algorithms, it not only has better real-time performance and accuracy, but also has better robustness to various environmental changes, and can recognize the types and important parts of various ships.